1. Field of the Invention
The present invention relates to a technique for improving the precision and stability of viewpoint position and orientation measurement.
2. Description of the Related Art
In recent years, studies about mixed reality (MR) that aims at seamless merging of physical and virtual spaces have been extensively made. An MR image is generated by superimposing and rendering virtual space images generated according to the position and orientation of an image sensing device such as a video camera or the like on a physical space image sensed by the image sensing device. An image display apparatus used in an MR system is implemented by, e.g., a video-see-through system. Note that the virtual space images include a virtual object rendered by computer graphics, text information, and the like.
In order to implement the MR, the accuracy of registration between the physical space and virtual space is important, and many approaches have been made conventionally in regard to this issue. A problem about registration in the MR eventuates in a problem of calculating the relative position and orientation between a target object on which virtual information is to be superimposed, and the image sensing device (to be referred to as the position and orientation of the image sensing device hereinafter).
As a method of solving this problem, the following attempt has been made. That is, a plurality of indices whose allocations on a target coordinate system are known are placed or set in an environment or on a target object. Then, the position and orientation of the image sensing device with respect to the target coordinate system are calculated using the three-dimensional (3D) coordinates on the target coordinate system of the indices as known information, and the coordinates of projected images of the indices in an image sensed by the image sensing device (see Sato and Tamura: “A Review of Registration Techniques in Mixed Reality”, Meeting on Image Recognition and Understanding (MIRU2002) Transactions I, IPSJ Symposium Series, vol. 2002, no. 11, pp. I.61-I.68, July 2002).
Also, an attempt that attaches an inertial sensor on an image sensing device and uses the sensor measurement value to achieve more stable registration than a case using only image information has been made. For example, a method that uses the position and orientation of the image sensing device estimated based on the sensor measurement value in index detection processing has been proposed. Also, a method that uses the estimation results as initial values for the position and orientation calculation based on an image has been proposed. Furthermore, a method that uses the estimation results as a rough position and orientation even in a situation in which indices are not observed has been proposed (see Japanese Patent Laid-Open No. 2005-33319, and Hirofumi Fujii, Masayuki Kanbara, Hidehiko Iwasa, Haruo Takemura, and Naokazu Yokoya, “A Registration Method Using Stereo Cameras with an Gyro Sensor for Augmented Reality”, Technical report of IEICE PRMU99-192 (Technical Report of IEICE, vol. 99, no. 574, pp. 1-8)).
The conventional registration technique using image information is premised on that all index detection results are correct. Furthermore, all index detection results are handled as even. For this reason, correct position and orientation measurement often fails due to the large influence of indices as detection errors or those with low detection precision.
Hence, the following technique has been proposed in recent years. That is, a statistical estimation method such as M estimation is adopted to calculate errors (re-projection errors) between the observation coordinates of the detected indices (feature points) on an image and the image coordinates (re-projected coordinates) of indices estimated from the position and orientation of the image sensing device and the positions of indices. Then, the reliabilities of the detected indices are calculated based on the errors to eliminate erroneously detected indices or to reduce their influences (see Sato, Kanbara, Yokoya, and Takemura, “Camera Movement Parameter Estimation from a Long Image Sequence by Tracking Markers and Natural Features”, Transactions of IEICE, D-III, vol. J86-D-II, no. 10, pp. 1431-1440, 2003).
However, the attempt that calculates the reliabilities based only on the statistical amounts of the re-projection errors, and weights the indices to eliminate detection errors or to reduce the influences of indices with low detection precision (the precision of image coordinates to be detected) is not always effective. This is because the above technique is effective to eliminate erroneously detected indices which appear exceptionally when many indices are detected correctly. However, when the number of erroneously detected indices is larger than that of correctly detected indices, the position and orientation measurement may be affected by indices with low detection precision. Since even indices sensed in a single image may often include those with low detection precision depending on the allocations of markers and the conditions upon image sensing, such indices lead to the drop of the position and orientation measurement precision.